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ABSTRACT 


The current S-Curve method of cost risk analysis for major DON acquisitions projects 
does not accurately estimate actual cost when the program reaches Full Rate Production. 
Another, sometimes more effective method of measuring cost risk, is by using the 
enhanced scenario-based method (eSBM) of risk analysis. The reason that cost 
estimations from the milestone B costs are inaccurate is that very little, if any, real 
information about the project is known. eSBM allows managers a less statistically 
tasking method of determining cost risk for a project while still maintaining the 
requirements of the Weapons System Acquisitions Reform Act. The key factors in 
measuring the usefulness of eSBM should be focused on the acquisition strategy being 
used for the project and the timeframe from Milestone B to later Milestones. I presume 
that different acquisition strategies will yield different levels of success in estimating cost 
risk for eSBM. 
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I. 


INTRODUCTION 


A. BACKGROUND 

The current method of choice for developing risk-adjusted cost estimates is the 
Monte Carlo simulation method, which sums probability-based cost distributions at the 
work breakdown structure (WBS) element level to produce a cumulative distribution for 
overall program cost. The confidence level from the cumulative distribution is the 
resulting “risk” for the estimated cost. This Monte Carlo method is also referred to 
within the cost-estimating community as the s-curve method of cost-risk analysis, and 
although it is the current method of choice for developing risk-adjusted cost estimates, it 
leaves several of the following problems unmanaged: 

1. Programs have not behaved as the cost estimation model had forecast. The 
cost risk becomes a second-order guess based on another guess. 

2. This model does not bound possible risk areas both on the high and low 
likelihood probability. By using this method, the left and right tails of the 
cumulative distribution are weighted equally with the mean. 

3. The model assumes that the consequences for all unfavorable outcomes 
equally affect the cost model. 

4. The model requires in-depth knowledge of statistical analysis and statistical 
processes. 

The s-curve method of cost-risk analysis for major Department of Defense (DoD) 
and Department of the Navy (DoN) acquisitions projects, so-called major defense 
acquisition programs (MDAPs), does not accurately estimate actual cost from the pre- 
Milestone B stages of the acquisitions process to the phase at which the program reaches 
full-rate production; thus, the s-curve method produces unreliable cost risk analysis. 

One of the reasons that cost estimates at Milestone B are inaccurate is that very 
little, if any, real information about the project is known other than the conceptual 
capabilities required. 
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The enhanced scenario-based method (eSBM) makes risk assessments without the 
need for details about the program, while still being grounded in historical facts and data. 
eSBM allows managers a less statistically taxing method of determining cost risk for a 
project. 


1. Purpose 

The key factors in measuring the usefulness of eSBM should be to focus on the 
acquisition strategy being used for the project and the time frame from Milestone B to 
later milestones. In this thesis, I hypothesize that different acquisition strategies will 
yield different levels of variation in estimating cost risk for eSBM. Furthermore, I 
hypothesize that the longer a program takes to reach full-rate production from the 
Milestone B, the more cost risk there is for that program. 

The eSBM method is not meant to replace use of statistical analysis to determine 
cost risk; it is intended to provide an alternative to the current standard. By having an 
additional method of risk analysis, cost estimators and program managers are provided 
with yet another risk profile with which to compare and measure risk. 

2. The Question 

There is no definitive research that links the effects of acquisition strategy and/or 
program length to program cost risk and the Monte Carlo method of cost-risk analysis 
makes no attempt to factor in acquisition strategy or the time horizon for the program. 
On the other hand, the eSBM makes an effort to reduce statistical uncertainty in 
estimating costs and reducing the risk of cost overrun. However, it falls short in the areas 
of acquisition strategy and time horizon. I hypothesize that different acquisition 
strategies will yield different levels of risk in estimating cost risk for eSBM. 

3. Who Benefits 

More accurate risk analysis benefits every level of DoD decision-making. The 
following examples outline some of these benefits. 

• By providing an accurate assessment of a program’s cost risk, the senior 

acquisition managers and Milestone Decision Authorities (MDA) will be 
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able to manage program risk early on and avoid potentially overly risky 
programs. Having another cost-risk profile will allow for more rational 
debates during the analysis of alternatives (AOA) stage of the acquisition 
process. By weeding out programs with a high likelihood of failure, more 
funds will be available for other programs that are likely to succeed. 

• Program executive offices (PEOs) and program managers (PMs) will be 
able to focus attention on areas of highest risk sooner in the program, thus 
detecting the potential effects on cost, schedule, and performance before 
problems arise. The more accurate cost-risk assessment would also allow 
for realistic decisions about how to manage the program as a whole. 

• Cost estimators would be provided a method of analysis that is founded in 
historical data without the rigorous statistical analysis. Due to the 
complex statistical analysis methods needed to produce accurate Monte 
Carlo simulations, cost estimations are subject to additional scrutiny from 
statisticians. The eSBM method removes this statistical scrutiny while 
still maintaining the desired outcome in terms familiar to decision-makers. 

In a dynamic world of fiscal constraints, accurate assessment of cost-estimating 
risk is even more crucial. 

4. The Next Steps 

In the next chapter, I describe how the eSBM was developed, how its parameters 
were derived, and how it is used for cost-risk analysis. In that section, I include a 
description of the methods used to construct the coefficients of variation (CV) for the 
different cost drivers. Additionally, I explain how to translate the eSBM results into 
terms familiar to the acquisition community. 

In this thesis, I present my research of how acquisition strategy and program 
length affect the cost-risk profile of MDAPs. This research consists of deriving the cost 
growth factors (CGF) and coefficients of variation by using historical data. After the 
CVs are formulated, I tested them for validity and translated into a cost risk profile. 
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II. LITERATURE REVIEW 


A. INTRODUCTION 

In 2005, Paul Garvey introduced the scenario-based method (SBM) as an 
alternative to the s-curve method of risk-adjusted cost estimation; he described it as an 
effective, less statistically intensive method of determining cost-risk analysis (Garvey, 
2005). The Air Force introduced the SBM in the 2007 Cost Risk and Uncertainty 
Handbook as an acceptable method to calculate cost estimate uncertainty (Air Force Cost 
Analysis Agency [AFCAA], 2007). 

In order to meet the requirements of the Weapons System Acquisition Reform Act 
(WSARA, 2009), the SBM would need to be able to produce confidence interval data that 
could be compared in the same manner as the s-curve analysis. Additionally, the new 
method would need to be coupled with historical data from previous defense acquisition 
programs. These requirements have led to the development of the statistical SBM, which 
uses simple algebraic formulas and basic statistical processes to generate cost risk curves. 
Flynn and Garvey conducted research on historical cost data to produce historically 
driven CVs, lending historical credibility to the statistical SBM. This historical data led to 
the eSBM. 

B. WEAPONS SYSTEM ACQUISITION REFORM ACT 

In 2009, the United States Congress passed the WSARA into law in an effort to 
provide better oversight and direction in the acquisition of defense programs. This law is 
intended “to improve the organization and procedures of the Department of Defense for 
the acquisition of major weapon systems” (Weapons System Acquisition Reform Act 
[WSARA], 2009, Introduction). With respect to cost estimation, the new law imposes 
two important requirements: 

1. The Director of the Cost Assessment and Program Evaluation (CAPE) shall “issue 
guidance relating to the proper selection of confidence levels in cost estimates 
generally, and specifically, for the proper selection of confidence levels in cost 
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estimates for major defense acquisition programs and major automated information 
system programs” (WSARA, 2009, § 101, subsection 2334(a)). 

2. The director shall also “disclose in accordance with paragraph (2) the confidence 
level used in establishing a cost estimate for a major defense acquisition program or 
major automated information system program, the rationale for selecting such 
confidence level, and, if such confidence level is less than 80 percent, the justification 
for selecting a confidence level of less than 80 percent” (WSARA, 2009, § 101, 
subsection 2334(d)). 

These two requirements drive the need to alter the SBM of cost estimation into a 
more statistically driven method. I discuss the basic form of the SBM and the 
enhancements needed to comply with the WSARA in Sections C and D of this chapter. 

1. Non-Statistical SBM 

The non-statistical SBM method analyzes the risks of various aspects of a project 
in a realistic manner in the opinion of the program manager and the experts conducting 
the analysis. This method brings to light all possible and reasonable risks for a program 
and discards the extreme outlier scenarios. Figure 1 illustrates the non-statistical SBM, 
and in the subsequent paragraphs I fully explain each step in in the method. 


Non-statistical SBM 



PS 


Figure 1. A Non-Statistical SBM (From Flynn and Garvey, 2011, p.7) 


The first step in this method is to add up estimates of all the elements from the 
work breakdown structure (WBS) as the program is defined in the program manager cost 
analysis requirements description (CARD). This sum of all the elements of the WBS is 
not adjusted for risk or uncertainty. The cost estimated from this step is called the 
program’s point estimate (PE). 
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The second step is to define the program’s protect scenario (PS). In this step, 
managers and analysts define the sources of major known areas of risk or uncertainty that 
have the potential to impact the cost estimate. This PS should not include worst-case or 
unrealistic scenarios but should include the areas in which managers should focus their 
attention. Once the PS is accepted, the cost associated with the scenario is estimated to 
determine the program’s PS cost. 

The difference between the PE and the PS costs is defined as the program’s cost 
reserve (CR), as illustrated in Equation 1. By increasing the program’s PS cost, the 
amount of CR also increases. Conversely, if the PE increases, the CR will be reduced by 
the same amount. 

CR = PS Cost- PF (1) 

Analysts are then able to conduct a sensitivity analysis of the PS cost and CR in 
order to improve the model. Once the model has been completed, the program office and 
cost estimators can finalize all estimates and generate all of the required reports. 

2. Statistical SBM 

The statistical SBM expands the non-statistical method by using the original PE 
and the addition of two inputs. These inputs are that the probability PE will not be 
exceeded [(a'.PR y and the coefficient of variation of program cost. By using the 
outputs from the non-statistical SBM, basic algebra, and simple statistical analysis, the 
basic SBM can be transformed to produce a cumulative distribution curve similar to the 
traditional s-curve methods. 

The analysts provide the probability that the PE cost will not be exceeded 
E(«I_PE ) and the coefficient of variation (CV) of program cost. The probability that the 
PE will not be achieved is based on the historical data and judgment of the PM and the 
cost estimators. The CV is based solely on the historical data from previous programs. 
Figure 2 shows the steps of the statistical SMB process. 
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Inputs for Statistical SBM 

• Point estimate (>-vr) 

• Probability PE will not be exceeded PE ) 

• Coefficient of variation (CV) 


Statistical SBM 


These steps are the same as the non-statistical SBM process 



Historical Data 
Guidelines 


These steps are specific to the statistical SBM process 
Inputs a PE and the coefficient of variation (CV) are specific to the statistical SBM process 


Figure 2. Statistical SBM (From Flynn and Garvey, 2011, p.9) 


The CV of a probability distribution is the ratio of the distribution’s standard 
deviation to its mean. Equation 2 illustrates the calculation for determining a CV. The 
CV lets cost estimators and analysts produce the probability distribution function (PDF) 
and cumulative distribution function (CDF). 

<7 

CV = P = - 

^ ( 2 ) 

There are two variations to the statistical SBM to account for the difference 
between normal and lognormal probability distributions (Flynn and Garvey, 2011, p.10- 
11). Equations 3-6 illustrate how the statistical SBM inputs are used to calculate the 
program’s mean cost and cost standard deviation. 

Normal Probability Equations 

D Xjjj j 

^ " Z,I l+P(i re ) (3 ) 
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Lognormal Probability Equations 
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( 6 ) 


The lognormal mean and standard deviation require conversion from log-dollars 
to dollars by Equations 7 and 8 (Garvey, 2005). 



(7) 


( 8 ) 


Analysts are now able to conduct a sensitivity analysis of the model by altering 
the ftpi and the CV. Alterations can create steeper or shallower slopes to the CDF, as 
well as shift the mean of the function. These alterations to the SBM allow cost estimators 
to produce reports that are similar to the widely accepted s-curve outputs. 

C. NCCA STUDY AND THE ENHANCED SBM 

In Section C, I demonstrated the methods necessary to produce confidence 
intervals and CDFs for the SBM. The alteration to the «si and the CV allows cost 
estimators to produce reports that are similar to the widely accepted s-curve outputs. The 
issue of how to ground the SBM in historical data still remains problematic. 

In 2011, Flynn and Garvey conducted research of historical costs across all realms 
of DoD acquisitions in order to integrate historical data into the SBM. The research 
focused on determining the appropriate value for the program’s and CV. 

The Flynn and Garvey study derived a value of «?£ -0.34 for programs at 
Milestone B based on a historical CV equal to 0.51 (Flynn & Garvey, 2011). Similar 
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findings were noted in the 2007 RAND report, which noted a = 0,09 based on an 
implied CV = 0.26 at Milestone B (Younossi, 2007). These two reports give analysts a 
bounded region of «pe to work within. Now that a historical value range for the upi has 
been derived, the only input left to consider is the CV. 

In the Flynn and Garvey study, research focused on determining what actually 
drives the value of the CV for various programs. Their first conjecture was to determine 
whether cost growth factors (CGFs) have been historically consistent. The researchers 
found routine underestimation of cost within the DoD. The second conjecture was that 
CVs decreased throughout the acquisition process. The researchers confirmed this 
assumption of decreasing CVs as programs progressed through the acquisition life cycle. 
The third conjecture was that CVs were equivalent for all acquisition programs regardless 
of platform. Again, the researchers proved this assumption to be true, especially from 
early milestones, such as Milestone B. The forth conjecture was that CVs would be 
lower if they were adjusted to account for quantity changes and actual inflation. Again, 
the NCCA researchers supported this assumption based on historical data. The fifth 
conjecture was that duration would have no effect on the CV. The researchers found no 
evidence to support this assumption; the data actually suggests quite the opposite (Flynn 
& Garvey, 2011). 

D. SUMMARY 

The possibility of producing historically based cost estimates with minimal 
statistical analysis has driven the desire for a new method of cost estimation. For any 
new method to be useful in the cost-estimating community, it would, as a minimum, have 
to meet the requirements of the WSARA. The non-statistical SBM produces a cost 
estimation method that is bounded by a scenario agreed upon by the program 
management team; it also derives the program’s PE. The program management team’s 
PE is coupled to a probability of achieving the estimated PE and a coefficient of variation 
to allow for statistically adjusted cost estimates. 
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During Flynn and Garvey’s study of program cost estimates, the eSBM gained the 
historical perspective necessary to achieve credibility with other cost estimators. The 
evolution of the SBM into the eSBM has revealed an all-new possibility of risk-adjusted 
cost estimation. 
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III. METHODS OF ANALYSIS 


A. NCCA HISTORICAL DATABASE 

1. How the Database Was Created 

The cornerstone of any research project is the collectable data. The NCCA 
collected and organized raw data from 100 Selected Acquisitions Reports (SARs). The 
SARs were mostly historical DoN major acquisition programs that were at various stages 
of completion (Flynn & Garvey, 2011). The database organized cost data in current-year 
and then-year values, quantity adjustments, information on platform type, milestone cost 
estimates, and annual program cost updates. By creating a well-functioning database, the 
groundwork for further analysis had been established. 

2. Assumptions Used in the Database 

The SAR is the best source of data for a program from a managerial standpoint. 
The two key cost estimates present in the SAR are the Baseline Estimate (BE) and the 
Current Estimate (CE). Although the cost estimates in the SAR are not perfect, numerous 
cost studies have indicated that the SAR is the best source for comprehensive program 
data (Flynn & Garvey, 2011, p.21). For the NCCA research as well as follow-on 
research, the BE is used as the initial cost estimate, and the CE of the last SAR for a 
program is used as the final actual cost. As stated by Flynn and Garvey (2011) in the 
eSBM, 

based on an analysis of 10 programs in our database dating from the 

1990s, there is little difference between the SAR BE, the program office 

estimate, and the Independent Cost Estimate (ICE) conducted either by the 

Naval Center for Cost Analysis (NCCA) or the Office of the Secretary of 

Defense (OSD), (p. 21) 

B. NCCA ANALYSIS METHOD 

In order to determine the CGF, the NCCA study used the ratio of BE to CE. As 
stated in Chapter II Equation 2, the CV is the ratio between the sample standard deviation 
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and the sample mean. For the different hypotheses in the NCCA study, the mean and 
standard deviation for each set of CGF was calculated to determine the applicable CV. 

1. Acquisition Milestone Adjustment 

In the NCCA study, researchers calculated the CGF at Milestones A, B, and C for 
all programs in the database regardless of platform or type. They then plotted the results, 
which revealed a lognormal distribution of CGFs for both the Milestones B and C. There 
was insufficient data to determine the distribution for data at Milestone A. The 
researchers used statistical analogy to determine the approximate CVs for Milestone A. 
After calculating the CGFs for each program, researchers then calculated the CV for each 
program using both quantity and non-quantity adjustments for base- and then-year 
dollars. They then plotted the mean and range for CVs at Milestones A, B, and C to 
reveal lower CVs for the later milestones. The more accurate cost estimation can be 
explained by technology maturation as well as by clearer information about program 
parameters. It is noted in the NCCA study that the range of the CVs narrowed at the later 
milestones. Figure 3 is an illustration of the results of this analysis. 



Figure 3. Benchmark CVs by Acquisition Milestone—Quantity Adjusted in Then- 

Year Dollars (From Flynn & Garvey, 2011, p.28) 

2. Platform Homogeneity 

In the NCCA study, researchers also analyzed the difference in cost growth 

between platforms. For this part of the study, researchers separated the CGFs by program 

type. The type categories included ships and subs, aircrafts, missiles, electronics, and 

other. Researchers calculated the mean and variance for each category to allow for an 
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analysis across categories. The surprising finding of this portion of the study was that the 
CGFs across platforms and categories were consistent. At both Milestones B and C, the 
CGFs were within the 5% alpha confidence level. This finding proved that there is 
significant homogeneity between programs. Figures 4 and 5 illustrate the relationship of 
CGF to platform type at Milestones B and C. 
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Figure 4. Milestone B: CGF Means and Variances (Quantity Adjusted in Then-Year 

Dollars) (From Flynn & Garvey, 2011, p.24) 
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Figure 5. Milestone C: CGF Means and Variances (Quantity Adjusted in Then-Year 

Dollars) (From Flynn & Garvey, 2011, p.27) 


3. Quantity Normalization 

To account for changes in the quantity of items being procured, researchers in the 
NCCA study analyzed the CV first without adjustment for changes in quantity and then 
adjusted for changes in quantity. It was no surprise that the adjusted CVs were lower 
than the unadjusted CVs. It is the opinion of the NCCA researchers that “changes in 
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acquisition quantity from a program baseline are generally regarded as beyond the 
purview of the cost analyst” (Flynn & Garvey, 2011, p. 22). The researchers conducted 
the quantity adjustment in two manners to determine whether the adjustment method had 
any effect on the calculated CVs. First, researchers scaled the BE to account for the 
actual number produced or acquired; next, they normalized the CE to reflect the actual 
cost if the original quantity was delivered. The differences between the two methods 
proved to be insignificant; however, the differences between the quantity-adjusted and 
the non-quantity-adjusted CVs were significant. 

4. Inflation Normalization 

Although SARs are adjusted to inflation, the values used are predicted inflation 
rates, not the actual rates observed during the time period covered by the program’s life 
cycle. Inflation rates are promulgated by the OSD to ensure that there is a standard set of 
inflation values used across all areas of the DoD. The problem arises when the actual 
rates are drastically different than the predicted rates from OSD. An example of drastic 
differences between the two rates occurred during the Carter administration when the 
observed rate was nearly 10.7%, whereas the OSD was forecasting rates between 3% and 
4% (Flynn & Garvey, 2011). By the time the OSD began to change the rates, inflation 
was on the decline and the OSD rates over estimated inflation during the Reagan 
administration (Flynn & Garvey, 2011). The researchers of the NCCA study calculated 
CVs using the original inflation rates and also using the actual historical inflation rates. 
Once again, they confirmed the expected effect of lower CVs for the historical inflation- 
adjusted estimates. 

C. METHOD FOR ACQUISITION STRATEGY CV ANALYSIS 

For my analysis of the effect of acquisition strategy on the CGF and CV, I placed 
each SAR in a category corresponding to the type of acquisition strategy used to procure 
the item. Once the programs were separated into their respective acquisition strategies, I 
computed, for each category, its standard deviation and mean. From the mean and 
standard deviation, I was able to calculate the CV for each acquisition strategy. To 
ensure that the changes in the CGF and CV were not due to factors other than the 
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acquisition strategy, I adjusted the values for the BE and CE to account for changes in 
quantity and for historical inflation rates. 

D. METHOD FOR PROGRAM-LENGTH CV ANALYSIS 

To analyze the effect of program length on the CV, I measured each program’s 
total length and added the data to the NCCA database. I calculated the CGF for each 
program and then plotted the CGF against the program life span. Once the CGFs were 
plotted with respect to time, I determined the mean and standard deviation for the entire 
sample set so that I could derive an overall CV for program length. 

I also segregated the CGF into three general time frames. The time frames were 
short acquisition lengths (< 5 years), average acquisition lengths (sample average +/- one 
standard deviation), and long acquisition lengths (> one standard deviation longer than 
the average). For each category, I then had a CV calculated to determine a relationship 
between the program length and the expected CV. I adjusted the values for the BE and 
CE to account for changes in quantity and for historical inflation rates to ensure that the 
effects of the variation were due to program length and not inflation or quantity changes. 

E. SUMMARY 

The NCCA database of over 100 SARs, spanning the last 4 decades, is the single 
best resource of data on historical cost. The key assumption used in the NCCA study was 
that the SAR BE would be used as the program’s initial cost estimate, and that the CE 
would be used as the program’s final cost. To normalize the initial estimates, the BE 
would have to be adjusted to account for changes in quantity and changes in actual 
inflation from the estimated inflation rates used in the cost-estimation process. A ratio of 
the BE and the CE are used to calculate the program’s CGF. The NCCA researchers 
calculated the CV for various combinations of CGFs to determine the appropriate CV to 
use in the eSBM. Additional research still needs to be conducted regarding acquisition 
strategy and overall program length. Comparing the CGFs for the different acquisition 
strategies and program lengths will provide further insight into the drivers for program 
CVs. 
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IV. ANALYSIS 


A. INTRODUCTION 

In this section, I describe the findings for the analysis of program length and 
acquisition strategy. Section B, in which I discuss program length, is divided into three 
parts: all programs, completed programs, and on-going programs. Section C, in which I 
discuss acquisition strategy, illustrates the problems with correlating any one acquisition 
strategy with a CGF. 

B. PROGRAM LENGTH 

I conducted the data analysis for program length to determine a correlation 
between program length and CGFs. I grouped the programs into two different categories: 
completed and on-going. Additionally, I developed a third group of programs to include 
all programs. In the following sections, I discuss the findings for the claim that the 
longer the programs length, the more cost growth can be expected. 

1. All Programs 

I used the all programs group as the control to determine any difference between 
the findings of Flynn and Garvey and my follow-on research (2011). The sample data 
that I used included 50 Milestone B cost estimates. I redid the analysis for CGFs and 
CVs for all programs to verify Flynn and Garvey’s results. My results were the exact 
same as those illustrated in Flynn and Garvey’s 2011 paper on the eSBM. Their findings 
make a clear connection between the CGF and the quantity adjustment. Table 1 
illustrates the Milestone B CGFs and CVs for all programs. Additionally, the 
relationship between base-year and then-year adjustments illustrates the relationship 
between the CGF and inflation for the same programs. 
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Cost Growth Factors & CVs for All DON MDAPs at MS B; n = 49 



(Without Qty Adjustment) 

(Quantity Adjusted) 

Statistics 

Base-Year$ 

Then-Year$ 

Base-Year$ 

Then-Year$ 

Mean 

1.49 

1.86 

1.24 

1.36 

Standard 

Deviation 

0.94 

1.61 

0.44 

0.69 

CV 

0.63 

0.87 

0.36 

0.51 


Table 1. Cost Growth Factors and CVs at Milestone B (All Programs) 

I analyzed the relationship between the CGF and program length for all programs. 
I assumed that no further schedule or performance growth would occur for the programs 
that were still on going. I conducted a regression analysis on all 50 programs. The most 
effective regression was the linear regression for all four cases: non-quantity-adjusted 
base-year dollars and then-year dollars, and quantity-adjusted base-year dollars and then- 
year dollars. Figures 6 and 7 illustrate the linear regression for all four cases. Figure 6 
shows a slight correlation between CGFs and program length for the non-quantity- 
adjusted cases. The correlation is not as evident in Figure 7 for the quantity-adjusted 
case. 

The regression results for all four cases yielded insignificant results in 
determining a correlation between the CGFs and the programs length. The results were 
insignificant at the 90% level, thus leading me to reject the hypothesis that there is 
correlation between the two variables. That is, I concluded that there is no correlation 
between the two variables. The analysis also failed the F test and t test, confirming the 
lack of correlation. The /^-squared and adjusted-/?-squared results for the non-quantity 
regression had a minimal significance; however, once quantity adjustments were made, 
there was no longer any noticeable significance. These results deny the claim that 
program length has any effect on program cost growth. 
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Figure 6. All Programs Non-Quantity-Adjusted Linear Regression 
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Figure 7. All Programs Quantity-Adjusted Linear Regression 
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2. Completed Programs 

For the completed programs, the only assumption I made was that there were no 
outstanding costs for the completed programs. This is a fair assumption considering the 
amount of time that had elapsed for a majority of the completed programs. My first task 
was to conduct regression analysis of the CGFs for all completed programs, from the 
Milestone B estimates against the program length. I set up the analysis to conduct all the 
normal regressions to include linear, exponential, and logarithmic regressions to produce 
the most significant regression result. The linear regression produced the most 
significant results for both quantity- and non-quantity-adjusted values of base-year 
dollars and then-year dollars. Figures 8 and 9 illustrate the linear regressions between the 
program length on the x axis and the CGF on the y axis. Figure 8 suggests only minor 
significance to the claim that program length has some effect on the CGF; however, once 
the program was adjusted for quantity changes, the value for /^-squared was reduced to an 
insignificant value, as illustrated in Figure 9. The adjusted /Csquared values revealed an 
even lower significance of the relation between the CGF and the program length to claim 
that program length has an effect on program CGFs for completed programs. The non- 
quantity-adjusted values produced results with modest correlation but that were still not 
above the 90% confidence level. Once I ran the correlations for quantity-adjusted CGFs, 
the correlation became even worse: all four analyses failed both the F test and 1 test for 
significance. I rejected the claim that program length affects cost growth for completed 
programs at the Milestone B cost estimate. 
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Figure 8. Completed Programs Non-Quantity-Adjusted Linear Regression 



Figure 9. Completed Programs Quantity-Adjusted Linear Regression 
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The next analysis I conducted was to determine the average CGF and CV for the 
completed programs. Table 2 illustrates the CGFs and CVs for the completed programs 
from the Milestone B estimates. These values are consistent with Flynn and Garvey’s 
findings illustrated in Table 1 (2011). These findings continue to validate the claim that 
changes in program quantity play the most significant role in determining cost growth. 
The results of the completed programs produced more significant results than the results 
of all programs due to the removal of the on-going programs. 


Cost Growth Factors & CVs for Completed DON MDAPs at MS B; n = 35 



(Without Qty Adjustment) 

(Quantity Adjusted) 

Statistics 

Base-Year$ 

Then-Year$ 

E3ase-Year$ 

Then-Year$ 

Mean 

1.61 

2.10 

1.26 

1.40 

Standard 

Deviation 

1.07 

1.84 

0.48 

0.78 

CV 

0.66 

0.87 

0.38 

0.56 


Table 2. Cost Growth Factors and CVs at Milestone B (Completed Programs) 

3. On-Going Programs 

The results for on-going programs revealed a negative relationship between CGFs 
and program length. The negative cost growth for programs still on going is not expected 
and counter-intuitive to the results of completed programs. Programs with the most time 
remaining until completion have the lower CGFs. Also, the differences between non- 
quantity-adjusted and quantity-adjusted CGFs are not as obvious, since many of the 
programs have yet to be affected by quantity drawdowns. Additionally, many of the 
programs have not been in existence long enough for the effects of inflation to become 
relevant. Figures 10 and 11 show the negative relationship between the CGFs and the 
program length for on-going programs. The regressions for the on-going programs also 
show large standard errors and low adjusted 7?-squared values. 
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There is some significance at the 90% confidence level for all four cases; however, there 
are only 10 observations that reduce the reliability of these findings. Once again, the 
findings of this research contradict the claim that program length has any effect on cost 
growth. 
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Figure 11. On-Going Programs Quantity-Adjusted Linear Regression 


Table 3 shows the average CGFs and CVs for the on-going programs. Even for 
the on-going programs, the CGFs and CVs for the quantity-adjusted values are within the 
range of the completed program threshold. 
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Cost Growth Factors & CVs for On-Going DON MDAPs at MS B; n = 10 



(Without Qty Adjustment) 

(Quantity Adjusted) 

Statistics 

Base-Year$ 

Then-Year$ 

Base-Year$ 

Then-Year$ 

Mean 

1.25 

1.33 

1.27 

1.35 

Standard 

Deviation 

0.41 

0.46 

0.37 

0.46 

CV 

0.32 

0.35 

0.29 

0.34 


Table 3. Cost Growth Factors and CVs at Milestone B (On-Going Programs) 

C. ACQUISITION STRATEGY 

1. Types of Acquisition Strategies 

The analysis of acquisition strategy resulted in more questions and no definitive 
answers. The problem with relating any one acquisition strategy to a CGF is the 
complexity of the acquisition system and modern program management. Program 
managers use a mixture of contract types during the Milestone B and beyond the 
contracts negotiation stage. For the current on-going programs, each program uses a 
mixture of firm-fixed-price (FFP), fixed-price-incentive-fee (FPIF), cost-plus-fixed-fee 
(CPFF), cost-plus-incentive-fee (CPIF), and cost-plus-award-fee (CPAF) contract types. 
The problem of assigning acquisition strategy to a single contract type becomes more 
difficult when the issue of awards and incentive fees is taken into account. Awards and 
incentives are written into contracts at the beginning but not awarded until contract 
completion. 

2. Continuation of Research 

The next step in trying to determine how acquisition strategy affects cost growth 
is to break down cost growth for each contract type into individual CGFs. Once the 
individual CGFs have been determined, they can be weighted to account for the relative 
amount each contract contributes to the total program cost. Breaking down the programs 
to the next lowest level will allow for further insight into the drivers for cost growth. 
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D. SUMMARY 


To the 95th percentile confidence level, I have rejected the hypothesis that as 
program length increases so does the CGF. Furthermore, the findings of this research 
have confirmed the findings of the NCCA study on CGFs and historical CVs. The 
analysis of program length yielded no significant relationship between the program 
length and CGFs. The analysis of program length reiterated the effects of quantity 
changes on CGFs as a significant driver for cost growth. Additionally, the effects of 
inflation on program cost growth continue to plague the cost-estimation community. 
Neither the changes in quantity nor the changes in inflation rates are under the control of 
the cost estimators; however, these changes must be accounted for in the cost risk profile. 
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V. CONCLUSION 


A. SUMMARY 

In Chapter I, I discussed the basis for this research and the issue of risk-adjusted 
cost estimation using the Monte Carlo simulation method. The defining issues with cost 
estimation at the Milestone B include the uncertainty of a clear program direction, the 
difficult statistical processes, and the unbounded reality of possible consequences. These 
difficulties lead to the need for a new method of calculating risk-adjusted cost estimation. 
A new method would be beneficial to everyone involved in the acquisition process, from 
the MDA to the cost estimators. 

In Chapter II, Literature Review, I described Paul Garvey and Brian Flynn’s 
findings as a more effective method of risk-adjusted cost estimation (2011). This chapter 
started with the necessary requirements defined in the WSARA. The WSARA tasks the 
Director of CAPE with issuing guidance on the appropriate confidence intervals to be 
used and with justifying their use if that level is below the 80% threshold. Garvey’s 
SBM provided the framework for a simplified method of risk-adjusted cost estimation 
using the statistical SBM (2005). Flynn and Garvey’s study on CGFs and CVs led to 
enhancements in the SBM that aligned the new eSBM with the requirements of the 
WSARA (2011). 

Chapter III, Methods of Analysis, includes two parts: what has been done in the 
past, and what this thesis does differently. The NCCA study brought to light the issue of 
inaccurate inflation estimates and quantity adjustments as a significant reason for cost 
growth. The NCCA study also debunked the theory that there are secular differences in 
cost growth across program types. The methods used in this thesis were aimed at relating 
cost growth to program length and acquisition strategy. 

I based my analysis in this thesis on two hypotheses: first, that cost growth will 
increase as the programs’ lengths also increase; and second, that different acquisitions 
strategies will yield different rates of cost growth. I rejected the first hypothesis with a 
high degree of confidence because I noted no correlation between the program length and 
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CGF. My analysis of program length did, however, further support Flynn and Garvey’s 
findings that inflation and quantity adjustment are the most significant sources of cost 
growth (2011). During my analysis of the second hypothesis, I discovered that no single 
acquisition strategy is used in contracting an MDAP. The complexity of acquisition 
strategy made the second hypothesis impossible to validate in the manner that I 
illustrated. 

B. RECOMMENDATIONS 

I recommend two main areas for continued research in historical cost growth and 
potential cost drivers. 

The first major area that needs to be researched is in contract management and in 
the methods used as a possible cost driver. This area should include a breakdown of each 
program to the contract level to determine the cost growth for different contact types, and 
then the contract types across programs could be compared as a potential link to cost 
growth. 

The second area in which research could be beneficial is the effect of awards and 
incentives on cost growth. An analysis of what percentage of cost growth is accounted 
for in the awards and incentives could lend insight into how to more accurately estimate 
awards and incentives for MDAPs. 

Other areas where research is still needed in the risk-adjusted cost estimation 
arena include 

1. how the eSBM compares to the Monte Carlo simulation at later milestones 

2. how to account for and eliminate PM bias during the cost-estimation process. 
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